Papers with NAR models
A Study of Non-autoregressive Model for Sequence Generation (2020.acl-main)
Copied to clipboard
| Challenge: | Non-autoregressive (NAR) models generate all tokens in parallel, resulting in faster generation speed compared to autoregressive models. |
| Approach: | They propose to use knowledge distillation and source-target alignment to bridge the gap between NAR and autoregressive models in various tasks. |
| Outcome: | The proposed techniques can speed up NAR models in some tasks but not all . the proposed techniques reduce target token dependency while allowing for faster inference . |
Improving Top-K Decoding for Non-Autoregressive Semantic Parsing via Intent Conditioning (2022.coling-1)
Copied to clipboard
| Challenge: | Semantic parsing (SP) is a core component of modern virtual assistants like Google Assistant and Amazon Alexa. |
| Approach: | They propose a non-autoregressive (NAR) semantic parser that introduces intent conditioning on the decoder. |
| Outcome: | The proposed model reduces inference latency while maintaining competitive parsing quality. |
ELMER: A Non-Autoregressive Pre-trained Language Model for Efficient and Effective Text Generation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for text generation use auto-regressive (AR) methods, but inefficient inference is a problem. |
| Approach: | They propose an efficient and effective PLM to explicitly model the token dependency during NAR text generation. |
| Outcome: | The proposed model outperforms existing models on three text generation tasks while achieving 10 times faster inference speedup. |
Non-Autoregressive Machine Translation: It’s Not as Fast as it Seems (2022.naacl-main)
Copied to clipboard
| Challenge: | Efficient machine translation models are commercially important as they can increase inference speeds, reduce costs and carbon emissions. |
| Approach: | They compare NAR models with autoregressive models to evaluate their performance . they point out flaws in evaluation methodology and argue for consistent evaluation . |
| Outcome: | The proposed model is faster on GPUs, but slower under more realistic usage conditions. |
How Does Distilled Data Complexity Impact the Quality and Confidence of Non-Autoregressive Machine Translation? (2021.findings-acl)
Copied to clipboard
| Challenge: | Prior work suggests that distilled training data is less complex than manual translations. |
| Approach: | They propose to use sequence-level knowledge distillation to match autoregressive models' translation quality. |
| Outcome: | The proposed model can match translation quality of autoregressive models with distilled training data. |
Releasing the Capacity of GANs in Non-Autoregressive Image Captioning (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing non-autoregressive (NAR) models suffer from their inherent multi-modality problem. |
| Approach: | They propose an Adversarial Non-autoregressive Transformer for Image Captioning that improves model performance by modifying model structure to be compatible with contrastive learning. |
| Outcome: | The proposed model achieves 26.72 times faster than the autoregressive model on the MSCOCO dataset. |